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Abstract 


This  study  assessed  the  usefulness  of  response  latency  data  for 
biographical  inventory  items  in  enhancing  the  inventory’s  validity.  The  Armed 
Services  Applicant  Profile  (ASAP)  was  computer  administered  to  Navy  recruits, 
and  the  regular  score,  latency-weighted  scores,  and  measures  of  deviant 
latencies  were  obtained.  The  latency-weighted  scores  did  not  improve  the 
ASAP  ’  s  validity  in  pr-cHirring  si::  month  retention,  when  used  instead  ot  or  in 
addition  to  the  regular  score,  and  the  deviant  latency  measures  did  not 
function  as  suppressor  or  moderator  variables  to  increase  the  ASAP ’ s  validity. 
But  subgroups  of  items  with  differing  latencies  varied  systematically  in  their 
internal-consistency  reliability  (with  increased  reliability  for  subgroups 
with  shorter  latencies) ,  and  a  small  subgroup  of  items  with  moderate  latencies 
was  almost  as  valid  as  the  regular  score,  suggesting  that  latency  data  may  be 
useful  in  writing  and  selecting  inventory  items. 


Response  Latency  Measures  for  Biographical  Inventories 

Recent  theoretical  and  empirical  work  in  personality  and  social 
psychology,  coupled  with  the  advent  of  computerized  testing,  raises  the  real 
possibility  of  improving  the  validity  of  personality,  interest,  and 
biographical  inventories  by  administering  them  via  computer  and  using 
information  about  latency  of  responding  to  the  items  to  modify  conventional 
scoring  techniques. 

Response  latencies  on  personality  inventory  items  and  personality-trait 
adjectives  have  been  extensively  studied  since  the  1970s.  A  key  finding  is 
that  items  with  long  latencies  are  unstable:  the  responses  to  these  items 
tend  to  change  on  retest.  In  itemmetric  studies,  latencies  and  the  proportion 
of  changed  responses  (over  a  four-week  interval)  correlated  .21  to  .41  for 
MMPI  (Hathaway  &  McKinley,  1951)  items  (Dunn,  Lushene,  &  O’Neil,  1972), 
latencies  and  changed  responses  (over  a  one-week  period)  correlated  .36  for 
Personality  Research  Form  (PRF;  Jackson,  1984)  items  (Holden,  Fekken,  & 
Jackson,  1985),  and  latencies  and  changed  responses  (over  a  one-month  period) 
correlated  .49  for  Basic  Personality  Inventory  (BPI;  Jackson,  1989)  items 
(Holden  &  Fekken,  1990). 1  (But  in  an  itemmetric  study  that  used  changed 
responses  on  immediate  retesting,  latencies  and  the  Ambdex  index  [Goldberg, 
1963],  a  measure  of  instability,  correlated  -.05  [not  significant]  for  PRF 
items  [Rogers,  1973].)  In  experiments  on  individual  differences,  the  PRF 
items  that  each  subject  changed  on  retest  (immediately  in  one  experiment; 
after  a  one-week  period  in  the  other)  were  predicted  significantly  better  than 
chance  on  the  basis  of  which  items  had  the  longest  latencies  for  him  or  her 
during  the  initial  administration  (Fekken  &  Jackson,  1988) . 

Several  otherwise  divergent  conceptualizations  are  alike  in  suggesting 
that  long  latencies  for  inventory  items  reflect  difficulty  in  responding. 
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Some  of  the  conceptualization  are  based  on  item  characteristics,  and  others  on 
the  interaction  between  individual  differences  and  item  characteristics 
(Fekken  &  Jackson,  1988).  The  item  characteristic  conceptualizations  argue 
that  the  difficulty  comes  about  because  the  item  is  hard  to  understand-- 
unreadable,  ambiguous,  etc.  (e.g.,  Dunn  et  al . ,  1972;  Hanley,  1962).  The 
conceptualizations  concerned  with  the  interaction  between  individual 
differences  and  item  characteristics  contend  that  the  difficulty  arises 
because  (a)  the  person  has  trouble  in  applying  the  item  to  himself  or  herself- 
-the  item  may  deal  with  matters  that  are  unfamiliar  or  unknown  to  the  person, 
or  the  different  response  alternatives  to  the  item  may  appear  equally 
descriptive  of  him  or  her  (e.g.,  Kuncel,  1973;  Markus,  1977;  Rogers,  1974a, 
1974b)  or  (b)  the  items  may  arouse  emotions  (e.g.,  Gilbert,  1967;  Temple  & 
Geisinger ,  1990) . 

The  observed  link  between  the  response  latency  of  personality  items  and 
the  items’  instability  implies  that  the  items’  latencies  are  also  associated 
with  the  items’  validity,  especially  their  predictive  validity — the  items  with 
the  longest  latencies  not  only  being  the  least  stable  but  also  the  least  valid 
over  time.  The  findings  in  the  two  investigations  that  bear  on  this  issue  are 
inconsistent.  In  one  study  (Holden  et  al . ,  1985),  latencies  for  PRF  items 
correlated  -.22  with  a  co-  .urrenc  validity  criterion  (a  composite  of  self- 
ratings,  self-reports  on  an  adjective  checklist,  and  preference  ratings),  but 
in  a  second  investigation  (.Holden  &  Fekken,  1990)  ,  latencies  for  BPI  items 
correlated  -.11  (not  significant)  with  another  concurrent  validity  criterion 
(ratings  by  clinicians).  However,  the  findings  of  these  studies  may  be 
affected  by  the  dichotomous  format  of  the  items:  items  with  extreme 
endorsement  proportions  tend  to  be  more  stable  and  less  valid  (Goldberg, 
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1963).  Furthermore,  these  results  were  based  on  concurrent  criteria,  and  the 
consequences  of  the  items’  instability  would  be  more  pronounced  if  predictive 
criteria  were  used. 

The  purpose  of  the  present  study  was  to  determine  whether  the  findings 
about  the  connection  between  item  latencies  and  item  instability  can  be  used 
to  improve  an  inventory’s  validity.  More  specifically,  the  main  goal  was  to 
assess  whether  weighting  item  scores  on  the  basis  of  their  latencies  improves 
the  predictive  validity  of  the  inventory’s  total  score.  A  secondary  aim  was 
to  assess  whether  measures  that  reflect  the  extent  to  which  subjects’ 
latencies  are  deviant  function  as  suppressor  or  moderator  variables  to 
increase  the  validity  of  the  inventory’s  total  score.  The  notion  is  that 
deviant  latencies  reflect  an  unusual  pattern  of  responding  to  the  inventory, 
stemming  from  idiosyncratic  difficulties  with  certain  items,  poor  test-taking 
attitudes,  and  other  variables  that  attenuate  validity.  Hence  using  measures 

of  deviant  latencies  to  suppress  this  invalid  variance  or  to  exploit  their 

interaction  with  the  inventory’s  score  should  increase  validity. 

Method 

Overview 

One  hundred  and  twenty  items  from  the  Armed  Services  Applicant  Profile 
(ASAP) ,  a  biographical  inventory,  were  computer  administered  to  Navy  recruits 
(all  men),  and  the  subjects’  response  choices  and  response  latencies  were 

recorded.  The  regular  score  for  the  ASAP  and  three  kinds  of  latency-weighted 

scores  were  obtained:  (a)  regular  scores  for  subgroups  of  items  with 
different  latencies,  for  optimal  weighting  by  standard  multiple-regression 
methods;  (b)  item  scores  directionally  weighted  by  their  latencies  (i.e.,  less 
weight  for  items  with  relatively  long  latencies);  and  (c)  item  scores 
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nondirectionally  weighted  by  their  latencies  (i.e.,  less  weight  for  items  with 
either  relatively  long  or  relatively  short  latencies) .  Measures  of  deviant 
latencies  and,  for  exploratory  purposes,  other  suppressor/moderator  variables 
were  also  obtained.  Data  for  the  criterion,  retention  in  the  Navy  for  six 
months,  was  subsequently  secured. 

ASAP 

Background .  The  ASAP  (Trent,  Quennette,  &  Pass,  1989)  is  designed  to 
predict  the  adjustment  of  enlisted  personnel  to  military  service.  The  final 
version  of  the  ASAP  consists  of  two  50-item  alternate  forms  drawn  from  an 
initial  pool  of  170  heterogeneous  items  chosen  for  their  potential  relevance 
to  adjustment.  The  items  have  three  to  five  alternatives ,  and  the 
alternatives  are  separately  scored  with  weights  of  1  to  3  that  have  been 
empirically  derived  to  predict  retention  at  21  months  of  service.  The  items 
encompass  six  factors  (nondelinquency,  work  orientation,  work  ethic,  academic 
achievement,  social  adaptation,  and  athletic  involvement). 

The  inventory’s  predictive  validity  against  retention  criteria  has  been 
extensively  studied,  using  a  cohort  of  applicants  for  active  duty  in  all  the 
armed  services.  The  present  form  of  the  ASAP  or  earlier  forms  (with  50  to  130 
items)  correlated  .18  to  .20  with  retention  at  6  months  (T.  Trent,  personal 
communication,  August  1986),  .21  with  retention  at  21  months  (Trent  et  al., 

198  ,  and  .27  with  retention  at  36  months  (Trent,  1989).  The  internal- 

consistency  reliability  was  .71  to  .76  for  50-item  forms  (Trent  et  al . ,  1989, 
1990)  and  .77  for  125  item  forms  (T.  Trent,  personal  communication,  December 
1987)  . 

I  terns .  A  set  of  120  items  was  available  for  this  study.  Twenty-six  of 
the  other  50  items  in  the  initial  pool  had  been  dropped  previously  because 
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they  concerned  circumstances  beyond  the  respondent’s  control,  they  might 
involve  ethnic  or  social  class  bias,  they  were  intrusive,  or  they  asked  about 
the  type  of  high  school  credential  (T.  Trent,  personal  communication,  October 
27,  1988).  The  additional  24  items  were  eliminated  for  this  study  because 
they  duplicated  remaining  items. 

Minor  editorial  changes  were  made  in  the  120  retained  items  to  achieve  a 
consistent  format  and  to  eliminate  unnecessary  instructions  (e.g.,  "Pick  the 
main  one"),  and  the  items  were  arranged  in  random  order. 

Because  the  current  item  weights  for  the  ASAP  are  unavailable  for  some 
of  the  120  items  and  are  based  on  retention  for  21  months  rather  than  the  six- 
month  period  used  in  this  study,  new  item  weights  were  obtained,  using  the 
same  procedures  and  the  same  cohort  data  (N=13,172  to  26,857)  employed  in 
deriving  the  current  weights  (Trent  et  al . ,  1989;  M.  A.  Quennette,  personal 
comniunicat"i  rin ,  lamia'"'’  1  989).  bu*-  for-  a  six-month  period. 

In  brief,  a  modification  of  the  "horizontal  percent"  method  (Stead  & 
Shartle ,  1940)  for  deriving  empirical  weights  for  biographical  items  was 
employed.  The  percentage  ot  applicants  retained  was  computed  for  each 
alternative  for  the  90  items  common  to  the  two  original  130  item  forms  of  the 
ASAP.  The  distribution  of  these  percentages  was  trichotomized ,  and 
alternatives  with  percentages  in  the  top  third  were  given  a  weight  of  3. 
alternatives  in  the  middle  third  a  weight  of  2,  and  those  in  the  bottom,  third 
a  weight  of  1.  These  weights  were  also  given  to  the  alternatives  f  r  run 
items  unique  to  each  form.  An  exception  was  made  for  alternatives  indicating 
that  the  respondent  did  not  graduate  from  high  school:  these  alternatives 
were  assigned  a  weight  of  1,  regardless  of  their  actual  weight,  to  make  the 
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ASAP  independent  of  high  school  diploma  status  for  policy  reasons.  (Only  one 
of  the  120  items  had  an  alternative  that  was  altered  for  this  reason.)2 

Computer  administration.  The  paper-and-pencil  version  of  the  ASAP 
was  adapted  for  computer  administration  via  the  same  Hewlett  Packard  Integral 
Personal  Computer  used  in  the  Accelerated  Computerized  Adaptive  Testing--Armod 
Services  Vocational  Aptitude  Battery  system  (Tiggle  &  Rafacz,  1985).  The 
computer-adapted  version  of  the  ASAP  was  designed  to  be  as  close  as  possible 
to  the  original  one  in  all  important  respects. 

The  computer  keyboard  was  simplified,  consisting  of  numerical  keys  for 
entering  the  subject’s  identification  number;  keys  labeled  A,  B,  C,  D,  and  E 
for  response  choices;  an  Enter  key;  and  a  Help  key.  The  subject  chose  a 
response  and  recorded  it  by  pressing  the  Enter  key.  The  response  could  be 
changed  at  will  before  the  Enter  key  was  pressed.  After  the  Enter  key  was 
pressed,  the  next  item  was  presented,  and  the  subject  could  not  return  to  the 
previous  item  or  earlier  ones.  The  subject  could  seek  assistance  from  the 
proctor  by  pressing  the  Help  key  and  raising  his  hand. 

The  pertinent  instructions  follow: 

Read  each  question  and  all  of  its  possible  answers  carefully,  then 
select  the  one  answer  that  is  best  or  most  appropriate  for  you. . . .You 
should  work  quickly  but  be  as  accurate  as  you  can.  Your  answers  to  some 
of  these  questions  may  be  verified  for  accuracy  and  honesty . 

Subjects  were  not  informed  that  their  latencies  were  being  recorded. 

The  following  information  was  recorded  for  each  item: 

1.  The  response  choice. 

2.  The  number  of  times  that  the  response  was  changed. 
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The  latency  (in  hundredths  of  a  second)  between  the  time  that  the 
item  was  presented  and  the  Enter  key  was  pressed. 

4.  The  number  of  times  that  the  Help  key  was  used. 

ASAP  measures .  The  regular  ASAP  score  (the  sum  of  the  regular  item 
scores)  and  three  kinds  of  latency-based  ASAP  scores  were  secured.  The 
latency-based  scores  employed  standardized  latencies,  and  two  versions  of  each 
score  were  obtained.  (Items  for  which  the  subject  used  the  Help  key  were 
excluded  in  standardizing  the  latencies  and  in  the  latency-based  scores;  items 
for  which  the  subject  changed  his  responses  were  included  in  the 
standardization  and  in  the  scores  because  of  the  prevalence  and  relevance  of 
changed  items . ) 

One  version  of  the  scores,  using  double-standardized  latencies  to 
eliminate  the  main  effects  of  individuals  and  items  (e.g.,  Popham  &  Holden. 
1990),  reflected  conceptualizations  concerned  with  the  interaction  between 
individual  differences  and  item  characteristics. 

First,  items  were  standardized  to  eliminate  item  differences  associated 
with  readability,  ambiguity,  and  other  characteristics.  For  this  purpose, 
"interquartile  deviations"  were  computed:  (Actual  Latency-Sample  Median)/ 

Sample  Interquartile  Range.  This  nonparametr ic  procedure  (Tukey,  1977)  was 
employed,  instead  of  the  conventional  procedure,  to  reduce  the  effects  on  the 
standardization  of  the  extreme  skewness  in  the  latency  data.  Note  that  this 
linear  transformation  does  not  distort  the  real-time  character  of  the  latency 
data  (Pachella,  1974). 

Then,  using  these  interquartile  deviations,  each  subject's  latencies; 
were  standardized  to  eliminate  individual  differences  associated  with  reading 
speed,  reaction  time,  and  similar  characteristics.  For  this  purpose,  "double- 
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stnndardized  interquartile  deviations"  were  computed:  (Interquartile 
Deviation-Subject’s  Median)/Subj ect ’ s  Interquartile  Range. 

The  other  version  of  the  scores,  employing  single-standardized  latencies 
to  eliminate  the  main  effects  for  individuals,  reflected  conceptualizations 
linked  wich  item  characteristics.  Using  the  actual  latencies,  each  subject’s 
latencies  were  standardized  to  eliminate  individual  differences. 

"Interquartile  deviations"  were  computed:  (Actual  Latency-Subject’s 
Median)/'Subj  ect  ’  s  Interquartile  Range. 

The  three  latency-based  ASAP  measures  follow: 

1.  Item  subgroup  scores:  mean  regular  item  score  for  each  of  10 
subgroups  of  12  items,  the  subgroups  varying  in  their  latencies,  and  the  items 
in  the  subgroups  differing  from  subject  to  subject.  For  example,  Subgroup  1 
had  the  items  with  the  largest  interquartile  deviations  (the  longest 
latencies)  for  each  subject,  and  Subgroup  10  the  items  with  the  smallest 
interquartile  deviations  (the  shortest  latencies).  When  an  item  was  excluded 
for  a  subject  because  the  Help  key  was  used,  his  Subgroup  10  had  the  11  items 
with  the  smallest  interquartile  deviations.) 

Ten  groups  of  items  were  used  to  achieve  adequate  reliability  while 
permitting  an  examination  of  subsets  of  items  with  extreme  latencies.  The 
scores  were  intended  to  be  combined  by  multiple  regression  methods  that  weight 
the  scores  for  maximum  validity  in  predicting  the  retention  criterion. 

2.  Directionally  weighted  item  score:  the  m  an  of  the  item  scores  that 
are  directionally  weighted  by  their  items’  corresponding  latencies.  Each 
subject’s  items  were  classified  into  nine  categories  on  the  basis  of  the 
interquartile  deviations,  ranging  from  -.81  or  more  (the  shortest  latencies) 
to  2.00  or  more  (the  longest  latencies).  His  regular  score  for  each  item  was 
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shrunk  towards  the  mean  for  the  sample,  depending  on  the  extremeness  of  the 
item’s  latency  (items  with  very  large  interquartile  deviations,  indicative  of 
long  latencies,  were  shrunk  the  most)  and  the  distance  between  his  score  and 
the  sample  mean.  (See  Table  1.)  The  subject’s  shrunken  item  score  was 
calculated  as  follows:  Subject’s  Item  Score  +  Shrinkage  Rate  (Sample  Mean  - 
Subject’s  Item  Score).  For  example,  suppose  a  subject  had  an  interquartile 
deviation  of  -.2  and  a  score  of  1  for  an  item,  and  the  mean  item  score  was  3. 
His  directionally  shrunken  score  would  be  1  +  25%  (3  -  1)  ■=  1.5. 

Nine  categories  were  chosen  to  provide  a  sufficient  range  of  adjustments 
in  the  item  scores.  The  nine  represent  equal  intervals  for  the  interquartile 
deviations  (except  for  the  intervals  at  each  end — the  interval  of  2.00  or  more 
at  the  high  end  corresponds  to  an  "outside"  outlier;  Tukey,  1977). 3  The  rates 
of  shrinkage  for  the  intervals  were  in  equal  steps,  going  from  100%  for  the 
lowest  interval  to  0%  for  the  highest.  The  basic  rationale  for  this  weighting 
procedure  is  that  the  score  is  invalid  for  an  item  with  a  very  long  latency, 
and  hence  the  best  estimate  of  this  score  is  the  sample  mean.  Thus  the  longer 
the  latency,  the  more  the  item’s  score  is  shrunk  to  the  mean. 

3.  Nondirectionally  weighted  item  score:  the  mean  of  the  item  scores 
that  are  nondirectionally  weighted  by  their  latencies.  This  score  was 
computed  in  the  same  way  as  the  directional  score,  except  that  items  with 
latencies  at  either  extreme  (very  large  interquartile  deviations,  indicative 
of  long  latencies,  and  very  small  interquartile  deviations,  indicative  of 
short  latencies)  were  shrunk  the  most.  (See  Table  1.)  For  example,  again 
suppose  a  subject  had  an  interquartile  deviation  of  -.2  and  a  score  of  1  for 
an  item,  and  the  mean  item  score  was  3.  His  nondirectionally  shrunken  item 


score  would  be  1  +  50%  (3  -  1)  -  2.0. 
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This  measure  used  the  same  nine  intervals  as  the  directionally  weighted 
item  score,  and  its  rates  of  shrinkage  were  in  equal  steps  from  0%  to  100%  to 
0%.  The  rationale  is  also  similar:  the  score  is  invalid  for  an  item  with 
either  a  very  long  or  a  very  short  latency,  and  hence  the  more  extreme  the 
latency  is  in  either  direction,  the  more  the  item’s  score  is  shrunk  to  the 
mean . 

This  nondirectional  measure  is  ad  hoc,  included  on  the  basis  of 
preliminary  results  with  the  double-standardized  item  subgroup  scores,  which 
exhibited  trends  towards  lower  reliability  and  validity  for  scores  with  eithc 
extremely  short  latencies  or  extremely  long  latencies. 


Insert  Table  1  about  here 


Deviant  latency  measures.  Four  measures  of  deviant  latencies  were  also 
obtained  (all  excluded  items  for  which  the  subject  used  the  Help  key): 

1.  The  product-moment  correlation  (transformed  to  Fisher’s  Z)  between  a 
subject’s  actual  latencies  and  the  sample’s  median  actual  latencies.  This  is 
an  index  of  the  correspondence  between  the  subject’s  and  the  sample’s 
latencies . 

2.  The  absolute  difference  between  the  subject’s  and  the  sample’s 
median  interquartile  deviation  for  items.  This  is  an  index  of  the  deviation 
between  the  subject’s  and  the  sample's  average  latencies. 

3.  The  subject’s  interquartile  range  for  item  interquartile  deviations. 


This  is  an  index  of  the  variability  of  the  subject’s  latencies. 
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4.  The  number  of  subject’s  double-standardized  interquartile  deviations 
of  3.5  or  more.  This  is  an  index  of  outlying  latencies  (an  interquartile 
deviation  of  3.5  defines  a  "far  out"  outlier;  Tukey,  1977). 4 
Other  Measures 

Several  measures  were  included,  for  exploratory  purposes,  as 
suppressor/moderator  variables.  Two  were  also  used  in  screening  the  sample 
(Maximum  Number  of  Changes  Per  Item  and  Effort  on  Test) ,  attenuating  their 
value  as  suppressor  or  moderator  variables  to  some  extent. 

Two  measures  were  secured  from  the  computer  administration  of  the  ASAP 
(both  excluded  items  for  which  the  subject  used  the  Help  key): 

1.  Proportion  of  items  changed.  This  is  a  control  for  individual 
differences  in  latencies  associated  with  changes  in  response. 

2.  Maximum  number  of  changes  per  item.  This  is  an  index  of  test-taking 
attitudes . 

Several  measures  were  obtained  from  a  paper-and-pencil  questionnaire 
completed  at  the  end  of  the  testing  session: 

1.  Effort  on  test.  This  is  the  sum  of  the  standard  scores  for  three 
variables : 

a.  At  the  beginning  of  the  test,  how  hard  did  you  try?  ried  Very 
Little  (1)... Tried  Very  Hard  (4). 

b.  At  the  end  of  the  test,  how  hard  did  you  try?  Tried  Very  Little 
(1) . . .Tried  Very  Hard  (4) . 

c.  Overall,  how  hard  did  you  try  to  do  on  the  test?  Tried  Very 
Little  (1)... Tried  Very  Hard  (4). 

2.  Tiredness  during  test.  This  is  the  sum  of  the  standard  scores  for 


two  variables: 
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a.  How  tired  did  you  feel  at  the  beginning  of  the  test?  Extremely 
Tired  ( 7 )... Extremely  Rested  (1). 

b.  Overall  how  tired  did  you  feel  at  the  end  of  the  test? 

Extremely  Tired  (7) ...  Extremely  Rested  (1). 

3.  Computer  use.  This  was  derived  from  the  following  question:  Within 
the  last  year,  how  often  have  you  used  a  computer?  Never,  or  1  to  10  times 
( 1 ) . . . 3 1  or  More  Times  (4). 

In  addition,  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB;  U.S. 
Department  of  Defense,  1984)  Paragraph  Comprehension  scale  score  was  obtained 
from  the  subjects’  records.  (The  ASVAB  was  completed  when  the  subjects  applied 
for  enlistment.)  This  is  a  15-item  measure  of  reading  comprehension. 

Criterion 

The  criterion  was  completion  of  six  months  (i.e.,  180  days)  of  active 
service  (or  separation  for  "nonpej orative"  reasons  during  that  period: 
officer  commission,  breach  of  contract  by  the  service,  death,  or  early 
release),  calculated  from  service  entry  date.  This  operational  definition  of 
retention  is  adapted  from  the  one  used  in  previous  ASAP  research  (Trent  et 
al.,  1989). 

Procedures 

The  ASAP,  followed  by  one  or  more  experimental  cognitive  tests,  was 
computer  administered  to  groups  of  approximately  30  subjects  from  February  to 
May  1989.  The  questionnaire  about  test-taking  attitudes  and  related  matters 
was  completed  at  the  end  of  the  session.  The  ASAP  administration  took 
approximately  a  half  hour,  and  the  entire  administration  about  two  and  a  half 
hours.  The  testing  room  held  a  battery  of  34  personal  computers. 
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Sample 

The  sample  consisted  of  1,090  Navy  recruits  (all  men)  at  the  Recruit 
Training  Center  (San  Diego) . 

All  recruits  in  the  available  units  were  asked  to  volunteer  to 
participate  in  the  study,  but  recruits  who  were  not  in  the  SAM  or  TAR  programs 
(reservists  with  limited  active-duty  obligations)  were  given  preference,  (SAM 
and  TAR  recruits  were  not  part  of  the  study  population.)  The  recruits  were 
instructed  that  the  test  results  would  not  affect  their  subsequent  assignments 
or  become  part  of  their  official  records. 

The  ASAP  was  administered  to  a  total  of  1,493  subjects.  Forty-two 
subjects  were  eliminated  because  information  was  unavailable  for  most  or  all 
of  their  pertinent  variables.  An  additional  136  subjects  not  part  of  the 
study  population  were  excluded  for  one  or  more  of  these  reasons: 

1.  They  were  in  the  SAM  program  (no  subjects  were  in  the  TAR  program). 

2.  They  had  prior  military  service  (or  information  about  this  matter 
was  missing) . 

3.  They  took  the  ASAP  more  than  15  days  after  service  entry  (or  this 
information  was  missing) . 

4.  They  had  a  dominant  language  other  than  English.5 

The  remaining  225  subjects  were  eliminated  because  of  their  test-taking 
behavior  or  attitudes.  This  was  done  in  two  stages  for  the  1,315  subjects  not 
already  excluded.  First,  122  subjects  were  dropped  for  either  of  two  reasons: 

1.  They  used  the  Help  key  for  more  than  one  item. 

2.  They  reported  on  the  paper-and-pencil  questionnaire  that  they  tried 
"Very  Little"  in  the  testing  session  (either  at  the  beginning,  at  the  end,  or 
overall,  or  this  information  was  missing). 
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Second,  of  the  remaining  subjects,  103  were  eliminated  for  any  of  these 
reasons : 

1.  They  made  more  than  five  changes  in  their  responses  to  an  item. 

This  corresponded  to  an  interquartile  deviation  of  3.5  in  the  distribution  for 
this  variable — a  "far  out"  outlier; 

2.  They  had  a  maximum  double-standardized  interquartile  deviation  of 
10.98  or  more.  This  corresponded  to  an  interquartile  deviation  of  3.5  in  the 
distribution  of  maximum  interquartile  deviations — a  "far  out”  outlier. 

3.  They  had  a  minimum  actual  latency  of  2.21  seconds  or  less.  This  was 
the  latency  by  the  fastest  .5%  of  the  sample  to  the  item  with  the  shortest 
latencies,  a  criterion  for  improbably  short  latencies  associated  with 
premature  responding  (Jensen,  1985). 

Analyses 

Internal-consistency  reliability  was  estimated  by  Coefficient  Alpha  for 
the  regular  ASAP  score  and  the  directionally  and  nondirectionally  weighted 
item  scores  and  by  the  intraclass  correlation  (Shrout  &  Fleiss,  1979,  Case  1 
for  mean  ratings)  for  the  item  subgroup  scores.. 

The  product-moment  intercorrelations  among  the  variables  were  computed. 
(Paragraph  Comprehension  scores  were  unavailable  for  32  subjects,  and  the 
sample  mean  was  substituted  for  the  missing  scores.) 

A  series  of  regression  analyses  of  the  four  kinds  of  ASAP  measures  was 
carried  out  against  the  retention  criterion.  The  comparative  validity  of  the 
measures  was  appraised  from  the  zero-order  correlations  of  the  regular  ASAP 
score,  nondirectionally  weighted  item  score,  directionally  weighted  item 
score,  and  the  multiple  correlation  of  the  item  subgroup  scores. 
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The  incremental  validity  of  the  latency-based  measures  when  combined 
with  the  regular  ASAP  score  was  assessed  by  hierarchical  regression  analyses 
(Cohen  &  Cohen,  1983)  :  the  zero-order  correlation  for  the  regular  ASAP  score 
was  compared  with  the  multiple  correlation  for  the  regular  ASAP  score  plus  the 
latency-based  measure.  This  analysis  was  done  separately  for  each  latency- 
based  measure.  (In  the  analyses  for  the  item  subgroup  scores,  the  scores  were 
treated  as  a  set,  and  Subgroup  10  was  excluded  to  avoid  collinearity  between 
the  regular  ASAP  score  and  the  subgroup  scores.) 

The  ability  of  the  deviant  latency  measures  and  the  other  variables  to 
suppress  or  moderate  the  validity  of  the  regular  ASAP  score  and  the  latency- 
based  ASAP  measures  was  also  assessed  by  hierarchical  regression  analyses.  A 
suppressor  effect  was  evaluated  by  (a)  a  comparison  of  the  zero-order  or 
multiple  correlation  for  the  ASAP  measure  with  the  multiple  correlation  for 
the  ASAP  measure  plus  the  suppressor/moderator  variable  and  (b)  a  comparison 
of  the  corresponding  zero-order  correlation  and  partial-regression  weights  for 
the  suppressor/moderator  variable,  if  the  first  comparison  revealed  a 
significant  difference  between  the  two  correlations.  (When  suppression 
exists,  the  regression  weight  for  a  variable  falls  outside  the  boundaries  set 
by  its  zero-order  correlation  and  zero;  Cohen  &  Cohen,  1983).  These  analyses 
were  done  separately  for  each  ASAP  measure.  (In  the  analyses  of  item  subgroup 
scores,  the  ten  scores  were  treated  as  a  set.) 

A  moderator  effect  was  evaluated  by  a  comparison  of  the  multiple 
correlation  of  the  ASAP  measure  and  the  suppressor/moderator  variable  with  the 
multiple  correlation  for  the  two  variables  plus  their  product  term  (the  latter- 
representing  the  interaction  between  the  ASAP  measure  and  the  suppressor/ 
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moderator  variable).  In  common  with  the  suppressor  analyses,  these  moderator 
analyses  were  done  separately  for  each  ASAP  measure,  and  the  item  subgroup 
scores  were  treated  as  a  set. 

Results 

Retention  Criterion 

The  retention  rate  was  91.2%:  994  subjects  of  the  1,090  subjects 

completed  six  months  of  active  service  (or  separated  for  nonpej orative 
reasons),6  comparable  to  the  91.3%  retention  rate  for  the  same  time  period  in 
previous  ASAP  research  (T.  Trent,  personal  communication,  August  1986). 

The  reasons  for  the  96  subjects’  attrition,  based  on  the  Interservice 
Separation  Codes,  are  reported  in  Table  2.  The  major  reasons  were  fraudulent 
entry  (28.1%),  erroneous  enlistment  or  induction  (25.0%),  and  trainee 
discharge  (21.9%). 

Given  the  extreme  split  in  the  retention  criterion,  the  maximum  product- 
moment  correlation  with  it  is  .57  (McNemar,  1962). 


Insert  Table  2  about  here 


Reliability  of  ASAP  Measures 

The  internal-consistency  reliability  of  the  regular  ASAP  score  and  the 
latency-based  ASAP  measures  is  reported  in  Table  3.  The  reliability  of  the 
item  subgroup  scores  is  also  shown  in  Figures  1  and  2.  The  regular  ASAP  score 
and  the  nondirectionally  weighted  item  score  were  somewhat  more  reliable  than 
the  directionally  weighted  item  score.  The  reliability  was  .80  for  the 
regular  ASAP  score,  comparable  to  the  previously  reported  reliability  of  .77 
for  a  125-item  form  (T.  Trent,  personal  communication,  December  1987).  The 
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reliability  was  also  .80  for  the  directionally  weighted  item  score  (both  the 
double-standardized  and  single-standardized  versions),  but  the  reliability  was 
lower  for  the  nondirectionally  weighted  item  score:  .74  for  the  double- 
standardized  version  and  .72  for  the  single-standardized  version. 

The  reliability  was  also  lower  for  the  item  subgroup  scores:  .14  to  .31 
for  the  double-standardized  versions,  and  .10  to  .40  for  the  single- 
standardized  versions.  The  trends  for  the  two  kinds  of  scores  diverged 
markedly.  For  the  double-standardized  version,  the  reliability  was  noticeably 
lower  for  scores  at  both  extremes  (Subgroup  1,  rtt-.14;  10,  r  t=.23).  For  the 
single-standardized  version,  the  reliability  systematically  increased  from  the 
score  with  the  longest  latencies  (Subgroup  1,  £tt=.10)  to  the  score  with  the 
shortest  latencies  (Subgroup  10,  rtt=.40). 

Because  the  item  subgroup  scores  were  based  on  12  items,  a  relevant 
comparison  is  the  estimated  reliability  (using  the  Spearman-Brown  formula)  of 
.29  for  the  regular  ASAP  score  with  the  same  number  of  items.  None  of  the 
double-standardized  scores  had  appreciably  higher  reliability  whereas  the  two 
extreme  scores  had  appreciably  lower  reliability  (Subgroup  1,  rt  -.14; 

Subgroup  10,  rtt=.23).  In  contrast,  the  two  single-standardized  scores  with 
the  longest  latencies  had  appreciably  higher  reliability  than  the  .29  estimate 
(Subgroup  9,  rtt=.37;  10,  ru=.40),  and  the  two  scores  with  the  shortest 
latencies  plus  a  score  with  moderate  latencies  had  noticeably  lower 
reliability  than  this  estimate  (Subgroup  1,  r{t= .  10 ;  2,  r  =.15;  5,  rtt=.21). 


Insert  Table  3  and  Figures  1  and  2  about  here 
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Intercorrelations  of  ASAP  Measures  and  Retention  Criterion 

The  intercorrelations  of  the  ASAP  measures  and  the  retention  criterion 
appear  in  Table  3.  All  the  ASAP  measures  (double-standardized  and  single- 
standardized  versions)  were  highly  correlated.  The  regular  ASAP  score 
correlated  .98  to  .99  with  the  two  versions  of  the  directionally  weighted  ice 
score  and  .96  to  .97  with  the  two  versions  of  the  nondirectionally  weighted 
item  score.  The  double-standardized  versions  of  the  directionally  and 
nondirectionally  weighted  item  scores  correlated  .95  with  each  other  and  the 
single-standardized  versions  correlated  .93.  And  the  multiple  correlations 
were  .99  and  .97,  respectively,  between  the  double-standardized  versions  of 
the  item  subgroup  scores  and  the  same  versions  of  the  directionally  and 
nondirectionally  weighted  item  scores;  the  corresponding  correlations  were 
1.00  and  .99  for  the  single-standardized  versions. 

Comparative  Validity  of  ASAP  Measures 

Figures  3  and  4  show  the  correlations  of  the  ASAP  item  subgroup  scores 
with  the  criterion. 

All  the  ASAP  measures  had  the  same  level  of  validity.  The  regular  ASAP 
score  correlated  .17  with  the  criterion,  comparable  to  the  .18  to  .20 
correlations  with  six-month  retention  reported  previously  (T.  Trent,  personal 
communication,  August  1986)  .  The  correlations  for  directionally  weighted  and 
nondirectionally  weighted  item  scores  were  similar:  .16  for  the  two  versions 
of  the  directionally  weighted  score,  and  .15  and  .16  for  the  two  versions  of 
the  nondirectionally  weighted  score.  The  multiple  correlations  of  the  item 
subgroup  scores  were  also  similar:  .18  for  both  versions. 

The  correlations  for  the  individual  item  subgroup  scores  were  lower: 

.07  to  .14  for  the  double-standardized  version,  and  .07  to  .12  for  the  single 
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standardized  version.  Again,  the  trends  for  the  two  versions  diverged.  For 
the  double-standardized  version,  the  trend  was  curvilinear:  the  correlation 
was  appreciably  higher  for  a  middle  score  (Subgroup  6,  r=.14)  and  noticeably 
lower  for  the  scores  at  the  extremes  (especially  Subgroup  10,  r=.07).  No 
trend  was  apparent  for  the  single-standardized  version. 

A  relevant  comparison  for  these  findings  about  the  item  subgroup  scores 
is  the  estimated  validity  of  .10  (using  the  Spearman-Brown  formula)  of  the 
regular  ASAP  score  for  12  items.  Only  one  double-standardized  score  had 
appreciably  different  validity  (Subgroup  6,  r=.14).  None  of  the  single- 
standardized  scores  had  noticeably  different  validity  than  this  estimate. 


Insert  Figures  3  and  4  about  here 


Incremental  Validity  of  Latency-Based  ASAP  Measures 

The  multiple  regression  analyses  of  the  incremental  validity  of  the 
latency-based  ASAP  measures  are  summarized  in  Table  4.  None  of  the  measui.es 
significantly  (p>.05)  increased  the  multiple  correlation  with  the  criterion 
when  combined  with  the  regular  ASAP  score. 


Insert  Table  4  about  here 


Incremental  Validity  of  Suppressor/Moderator  Variables 

The  multiple  regression  analyses  of  the  incremental  validity  of  the 
suppressor/moderator  variables  are  summarized  in  Table  5.  None  of  these 
variables  significantly  (p>,05)  increased  the  multiple  correlation  with  the 
criterion  when  combined  with  the  regular  ASAP  score  or  with  the  latency-based 
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ASAP  measures,  indicating  that  the  suppressor/moderator  variables  were  not 
functioning  as  suppressor  variables. 

With  one  exception,  none  of  the  variables  significantly  (g>.05) 
increased  the  multiple  correlation  when  their  product  score  was  combined  with 
the  ASAP  measure  and  the  suppressor/moderator  variable,  indicating  that  the 
suppressor/moderator  variables  were  not  functioning  as  moderator  variables. 
The  exception  involved  the  Interquartile  Range  and  single-standardized  item 
subgroup  scores.  The  multiple  correlation  in  this  analysis  increased  to  .2-'* 
from  .19  when  the  product  score  was  added  (p<.05). 


Insert  Table  5  about  here 


Discussion 

Theoretical  and  Methodological  Implications 

It  is  apparent  from  the  results  that  the  latency-based  ASAP  measures  did 
not  improve  the  biographical  inventory’s  predictive  validity  when  used  instead 
of  or  in  addition  to  the  conventional  ASAP  score.  And  it  is  equally  clear 
that  the  measures  of  deviant  latencies,  along  with  the  exploratory  variables, 
did  not  function  as  suppressor  or  moderator  variables  to  enhance  the  ASAP’s 
validity  either.  (The  single  instance  in  which  a  deviant  latency  measure, 
displayed  a  moderator  effect--the  Interquartile  Range  vis-a-vis  the  single- 
standardized  item  subgroup  scores--is  not  readily  interpretable  and  probably 
represents  a  chance  outcome  of  the  large  number  of  significance  tests 
performed. ) 

Nonetheless,  some  important  positive  findings  did  emerge.  Consistent 
with  the  expectation  based  on  previous  results  that  items  with  long  latencies 
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are  unstable,  systematic  trends  in  reliability  occurred  in  the  analysis  of 
item  subgroups  with  single-standardized  latencies,  with  lower  internal- 
consistency  reliability  for  subgroups  of  items  with  longer  latencies.  The 
findings  were  much  less  clear  cut  in  the  reliability  analysis  of  item 
subgroups  with  double-standardized  latencies  but  suggested  lower  reliability 
for  subgroups  with  either  very  long  or  very  short  latencies.  This 
unanticipated  possibility  that  items  with  unusually  short  latencies  may  also 
be  unstable  needs  to  he  followed  up.  An  obvious  conjecture  is  that  verv  short 
latencies  indicate  subjects  are  paving  minimal  attention  to  the  item  content 
or,  at  wo  t,  are  responding  more-or-less  randomly.  The  sample  was  screened 
to  eliminate  subjects  with  poor  test-taking  attitudes,  including  individuals 
making  impossibly  fast  responses,  but  this  process  excluded  only  those  with 
extreme  behavior. 

The  trends  in  reliability  in  this  analysis  of  single-standardized 
latencies  support  and  extend  itemmetric  studies  that  uncovered  a  substantial 
association  between  latencies  and  instability  for  personality  items  (Dunn  et 
al.  ,  1972;  Holden  et  al.  ,  1988;  Holden  &  Fekken,  1990).  Because  the  single- 
standardization  procedure  was  used,  considerable  commonality  probable  exists 
in  the  items  that  make  up  the  item  subgroup  scores.  The  present  findings 
indicate  that  the  earlier  results  about  retest  reliability  also  apply  to 
internal-consistency  reliability  and  suggest  that  the  previous  findings  wore 
not  simply  an  artifact  of  the  dichotomous  character  of  the  personality  items 
(Goldberg,  1963).  The  failure  of  these  clear-cut  trends  in  reliability  to  be 
paralleled  by  similar  trends  in  validity  may  occur  because  of  the  generally 
low  level  of  validity  involved. 
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These  reliability  trends  have  implications  for  writing  and  selecting 
inventory  items.  First,  reliability  (and,  in  turn,  validity)  may  be  maximized 
by  writing  items  that  elicit  short  latencies.  Such  items  are  primarily  short 
in  length  and  unambiguous  (Dunn  et  al.,  1972;  Holden  et  al.,  1988). 

Second,  latency  data  may  be  useful  in  selecting  items  for  reliabilitv 
(and,  indirectly,  for  validity)  (Fekken  &  Jackson,  1988;  Holden  &  Jackson, 
1990) .  Standard  item  analytic  methods  that  choose  items  with  high 
correlations  with  the  total  score  or  the  criterion  can  accomplish  these 
purposes,  too.  But  latency  data  may  be  particularly  useful  when  (a)  the 
measure  is  heterogeneous  and  hence  item-total  score  correlations  are  nf 
questionable  value,  (b)  the  criterion  has  limited  validity,  or  (c)  the 
criterion  requires  time  to  mature. 

Another  important  finding  concerns  the  expectation  that  items  with  long 
latencies  are  less  valid.  The  findings  in  the  analysis  of  double-standardized 
item  subgroup  scores  suggested  that  items  with  very  long  latencies  as  well  ns 
those  with  very  short  latencies  were  less  valid.  Furthermore,  this  analysis 
identified  a  subset  of  items  with  moderate  latencies  (Subgroup  6)  that  were 
more  valid  than  the  other  sets  and  almost  as  valid  as  the  regular  ASAP  score. 
Indeed,  the  estimated  validity  of  this  subgroup  score  would  be  .24  (using  the 
Spearman-Brown  formula)  if  it  had  as  many  items  as  the  regular  ASAP  score, 
appreciably  larger  than  the  latter’s  validity  of  .17.  This  result  clearly 
needs  to  be  replicated,  but  it  offers  the  intriguing  prospect  of  improving  the 
ASAP ’ s  validity  by  using  more  of  the  same  kind  of  items  that  are  in  this 
subgroup .  Because  the  double-standardization  procedure  clustered  items  on  the 
basis  of  their  subject,  by  item  interactions,  it  is  unlikely  that  appreciable 
commonality  exists  in  the  items  in  this  subgroup.  Consequently,  it  would 
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probably  be  necessary  to  identify  the  appropriate  items  individually  for  each 
subject,  using  computerized  adaptive  testing.  How  accurately  such  items  can 
be  identified  remains  to  be  seen. 

One  other  outcome  is  noteworthy.  The  similar  validity  of  the  ASAP 
regular  score  and  the  latency-based  measures  indicates  that  the  unorthodox 
methods  used  to  devise  the  latter--the  item  subgroup  scores  that  rely  on 
comparable  scores  from  different  sets  of  heterogeneous  items,  and  the 
weighting  scheme  used  in  the  directionally  and  nondirectionally  weighted  item 
scores--did  not  degrade  the  ASAP’s  validity.  This  outcome  implies  that  fhese 
unusual  procedures  were  reasonable. 

All  in  all,  the  findings  for  the  item  subgroup  scores,  as  well  as  the 
results  for  the  weighted  item  scores,  offer  mixed  support  for  the  two  kinds  of 
competing  conceptualizations:  (a)  individual  differences  by  item 
characteristics  interaction,  represented  by  the  double-standardized  measures; 
and  (b)  item  characteristics,  represented  by  the  single-standardized  measures. 
The  most  clear-cut  confirmation  was  associated  with  the  reliability  findings 
for  the  single-standardized  item  subgroup  scores. 

It  should  be  recognized  in  this  connection  that  the  empirical  keying  of 
the  ASAP  items  hampered  the  ability  of  the  single-standardization  procedure  to 
improve  validity.  Insofar  as  the  subjects’  item  latencies  are  in  the  same 
order,  and  the  present  sample  is  comparable  to  the  one  used  in  deriving  the 
item  scores,  latency  data  will  not  improve  validity  because  the  items  already 
have  optimal  weights  for  predicting  the  criterion.  For  instance,  suppose  that 
the  items  with  the  longest  latencies  for  everyone  in  the  sample  were  also  the 
least  valid.  The  weights  for  the  items  reflect  the  level  of  validity  for  the 
sample,  and  adjusting  the  weights  in  the  same  way  for  each  subject  (because 
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all  subjects  have  the  same  latencies)  will  have  no  effect.  The  ASAP’s 
empirical  keying  does  not  affect  the  reliability  results  for  the  single- 
standardized  procedure  because  the  items’  weights  are  not  optimal  for 
reliability.  The  keying  also  does  not  affect  either  the  validity  or 
reliability  results  for  the  double-standardized  procedure  because  this 
standardization  makes  the  latencies  and  the  resulting  adjustments  different 
for  each  subject. 

The  findings  also  offer  no  basis  for  choosing  between  the  directionally 
and  nondirectionally  weighted  items,  for  the  two  performed  similarly:  their 
validity  was  equivalent,  though  the  directional  version  was  somewhat  more 
rel iable . 

Future  Directions 

Future  efforts  might  benefit  by  partitioning  response  latency  into  its 
major  components.  Rogers  (1974a;  1974b,  p.  130)  has  distinguished  three  main 
stages  in  responding  to  an  inventory  item:  stimulus  encoding,  stimulus 
comprehension,  and  binary  "true/false"  decision  about  the  item.  This  last 
stage,  in  turn,  is  divided  into  two  substages:  self-referent  decision 
("relating  of  the  internalized  item  content  to  the  ’self-concept’1')  and 
response  selection.  Similarly,  Kuncel  (1973,  p.  547)  has  delineated  two 
stages:  meaning  ("attributing  some  ’reasonable'  interpretation  to  the  irem") 

and  apply  ("employing  information  which  is  ’well-suited’  as  a  basis  for 
answering  the  item"). 

The  present  study,  in  common  with  most  research  in  this  rield,  measured 
and  used  the  total  time  involved  in  all  these  stages  of  responding,  but  the 
time  consumed  in  the  various  stages  may  have  very  different  psychological 
imp  1 ications .  The  conceptualizations  linked  with  item  characteristics  (e.g., 
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Dunn  et  al . ,  1972;  Hanley,  1962)  focus  on  the  early  stages  (e.g.,  Rogers’ 
stimulus  encoding  and  stimulus  comprehension;  Kuncel’s  meaning).  In  contrast, 
conceptualizations  concerned  with  the  interaction  between  individual 
differences  and  item  characteristics  (e.g.,  Gilbert,  1967;  Kuncel,  1973; 
Markus,  1977;  Rogers,  1974a,  1974b;  Temple  &  Geisinger,  1990)  primarily  deal 
with  the  later  stages  (Rogers’  binary  decision  stage,  Kuncel’s  apply  stage) 

As  a  first  step  in  partitioning  latencies,  it  would  be  prudent  to 
control  experimentally  for  item  length,  as  had  been  done  in  a  few  studies 
(Hanley,  1962;  Rogers,  1974a,  1974b;  Temple  &  Geisinger,  1990),  because  of  its 
generally  high  association  with  latencies  (Dunn  et  al . ,  1972;  Holden  et  al . , 
1985;  Tetrick,  1989). 

It  would  be  worthwhile  to  employ  a  more  predictable  criterion,  as 
suggested  earlier.  The  ASAP’s  validity  was  modest  against  the  six-month 
retention  criterion  in  this  study,  with  correlations  accounting  for  less  than 
4%  of  the  criterion  variance.  In  these  circumstances,  substantial  increases 
in  validity  with  improvements  in  the  ASAP  are  difficult  to  uncover,  even  when 
reliability  is  dramatically  enhanced,  as  in  the  analysis  of  the  single- 
standardized  item  subgroup  scores.  At  the  least,  a  36-month  retention 
criterion  might  be  used,  given  the  substantially  higher  validity  of  the  ASAP 
with  36-month  retention  (r— .27;  Trent,  1989)  than  with  6-month  retention 
(r=.18  to  .20;  T.  Trent,  personal  communication,  August  1986).  Other 
criteria  for  the  ASAP  might  also  be  employed,  such  as  disciplinary  records, 
promotions,  and  recommendations  for  re-enlistment. 

It  might  also  be  fruitful  to  investigate  situational  influences. 

Latencies  may  convey  more  significant  information,  and  consequently  latency- 
based  measures  may  be  more  useful,  in  situations  that  are  perceived  as  more 
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important  than  the  typical  research  study.  Investigations  that  use  the 
measures  in  actual  selection  or  in  faking  experiments  that  approximate  such 
demand  characteristics  are  especially  relevant.  It  is  noteworthy  that  a 
recent  faking  experiment  (Hsu,  Santelli,  &  Hsu,  1989)  found  that  latency 
measures  (the  mean  latencies  for  subtle  and  obvious  MMPI  items)  were  able  to 
detect  dissimulation. 

More  research  is  obviously  needed  on  methods  for  weighting  individual 
items  and  for  grouping  items  into  subsets  by  their  latencies.  The  schemes 
used  in  this  study  were  no  more  than  first  approximations,  and  a  variety  of 
improvements  are  possible. 

Finally,  a  systematic  appraisal  of  the  efficacy  of  latency  data  in 
writing  and  selecting  inventory  items  is  called  for.  Studies  that 
experimentally  manipulate  item  latencies,  via  changes  in  the  length  and  other 
characteristics  of  the  items,  and  then  compare  the  items’  reliability  and 
validity  av  ■>  pertinent.  Equally  relevant  are  investigations  that  assess  the 
relative  reliability  and  validity  of  items  selected  on  the  basis  of  latency 
data  with  items  chosen  by  standard  item  analytic  methods. 

Cone lus ion 

Although  this  initial  effort  at  using  response  latency  data  to  improve 
the  validity  of  a  biographical  inventory  directly  was  unsuccessful,  there  were 
strong  indications  that  employing  these  data  in  developing  an  inventory  may 
enhance  validity  indirectly,  and  thereby  accomplish  the  same  goal. 

It  should  also  be  borne  in  mind  that  closely  related  work  has  directly 
improved  the  validity  of  personality  inventories.  Several  recent  studies  have 
found  that  latency  scores  for  a  scale  (i.e.,  the  mean  latency  for  endorsed 
items  on  the  scale,  and  the  mean  latency  for  rejected  items  on  the  scale) 
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frequently  had  incremental  validity  in  predicting  external  criteria,  when 
combined  with  the  regular  scale  score  (Fekken,  1990;  Holden,  Fekken,  &  Cotton, 
in  press;  Mervielde,  1988;  Popham  &  Holden,  1990).  This  particular  approach 
requires  items  that  are  dichotomous  and  homogeneous,  and  hence  is  inapplicable 
to  a  heterogeneous  biographical  inventory,  such  as  the  ASAP.  Nonetheless, 
these  findings  underscore  the  potential  for  latency  data. 

Given  the  ease  of  collecting  response  latency  information,  its  ability 
to  improve  the  validity  and  utility  of  self-report  inventories  merits  serious 
investigation. 
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Footnotes 

^he  signs  of  the  correlations  in  the  Dunn  et  al .  and  the  Holden  and 
Fekken  studies  have  been  reflected  to  be  consistent  with  the  reversal  of  the 
dependent  variable  in  these  investigations  from  the  proportion  of  unchanged 
responses  to  the  proportion  of  changed  responses. 

2Several  of  the  current  weights  were  assigned  on  rational  grounds  to 
improve  content  validity;  it  was  not  feasible  to  replicate  that  process  for 
this  study. 

3An  interquartile  deviation  of  2.00  defines  the  "inner  fence"  of  a 
frequency  distribution  (Tukey,  1977). 

4An  interquartile  deviation  of  3.50  defines  the  "outer  fence"  of  a 
frequency  distribution  (Tukey,  1977). 

5Dominant  language  was  assessed  by  the  following  question  that  was 
computer  administered,  immediately  preceding  the  ASAP:  What  language  do  you 
read  and  write  best?  (A)  English,  (B)  Spanish,  (C)  Chinese,  (D)  Tagalog,  (E) 
Some  other  language. 

^Two  subjects  were  separated  for  nonpej orative  reasons:  Non-Battle 
Death — Other,  and  Death--Cause  Not  Specified. 
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Shrlnkage  Rates  for  Directionally  and  Nondirectionallv  Weighted  Item  Scores 


Shrinkage  Rate 


Interquartile  Deviation 

Directional 

Nondirect ional 

-.81 

or 

more 

0, 

.  0% 

100, 

,  0% 

-.80 

to 

-.41 

12, 

.5 

75, 

.0 

-.40 

to 

-.01 

25, 

.0 

50, 

,0 

.00 

to 

.39 

37, 

,5 

25. 

,0 

.40 

to 

.79 

50, 

.0 

0. 

0 

.80 

to 

1.19 

62 

.5 

25. 

.0 

1.20 

to 

1.59 

75, 

.0 

50, 

.0 

1.60 

to 

1.99 

87. 

.5 

75 

.0 

2.00 

or 

More 

100, 

.0% 

100, 

.0* 
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Table  2 

Reasons  for  Attrition,  N  -  96 

Reason3  Percent 

Medical  conditions  existing  prior  to  service  3.1% 

Medically  unqualified  for  active  duty — other  4 .2 

Character  or  behavior  disorder  8.3 

Drugs  1 . 0 

Fraudulent  entry  28.1 

Good  of  the  service  2  .  1 

Basic  training  attrition  2.1 

Trainee  discharge  21 

Erroneous  enlistment  or  induction  25.0 

Other  •'* .  2  t 


aIrrerservice  Separation  Code 


Table  4 


-37- 


IncremenCal  Validity  of  Latencv-Based 

ASAP  Measures 

when  Combined 

with  Reeular 

ASAP  Measures.  N  =  1.090 

Measures 

n 

R 

R  Increase 

Regular  ASAP 

i 

. 1653** 

Regular  ASAP  and  Directionally 
Weighted  Item  Score — Double 
Standardized 

2 

. 1681** 

.0028 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Nondirectionally 
Weighted  Item  Score--Double 
Standardized 

2 

.1667** 

.0014 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Item  Subgroup 
Scores--Double  Standardized 

10 

.1809** 

.0156 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Directionally 
Weighted  Item  Score--S ingle 
Standardized 

2 

. 1656** 

.  0CC3 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Nondirectionally 

Wf lghted  Item  Score--S ingle 
Standardized 

2 

. 1660** 

.0007 

Regular  ASAP 

1 

.  1653** 

Regular  ASAP  and  Item  Subgroup 
Scores--S ingle  Standardized 

10 

. 1826** 

.0173 

' p  <.01 


Table  5 
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Incremental  Validity  of  Suppressor /Moderator  Variables  when  Combined  with 
Regular  and  Latency-Based  ASAP  Measures.  N  -  1.090 


Measures 

n 

R 

R  Increase 

Regular  ASAP 

i 

.1653** 

Regular  ASAP  and  Correlation  Between 
Subject’s  and  Sample’s  Latencies 

2 

. 1709** 

.0056 

Regular  ASAP,  Correlation  Between 
Subject’s  and  Sample’s  Latencies, 
and  Product 

3 

. 1709** 

.0000 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Interquartile  Range 

2 

. 1728** 

.0075 

Regular  ASAP,  Interquartile  Range, 
and  Product 

3 

. 1728** 

.0000 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Interquartile 
Deviations  of  3.5  oi  More 

2 

. 1667** 

.  0019 

Regular  ASAP,  Interquartile 

Deviations  of  3.5  or  More,  and 

Product 

3 

. 1713** 

.0046 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Difference  Between 
Median  Interquartile  Deviation 
for  Subject  and  Sample 

2 

. 1656** 

.0003 

Regular  ASAP,  Difference  Between 

Median  Interquartile  Deviation 
for  Subject  and  Sample,  and  Product 

3 

. 1662** 

.0006 
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Table  5  (Continued) 


Measures 

n 

R 

R  Increase 

Regular  ASAP 

i 

. 1653** 

Regular  ASAP  and  Items  Changed 

2 

. 1669** 

.0016 

Regular  ASAP,  Items  Changed,  and 
Product 

3 

.1731** 

.0062 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Maximum  Changes 

Per  Item 

2 

. 1728** 

.0075 

Regular  ASAP,  Maximum  Changes  Per 

Item,  and  Product 

3 

. 1728** 

.0000 

Regular  ASAP 

1 

.1653** 

Regular  ASAP  and  Effort  on  Test 

2 

.1690** 

.0037 

Regular  ASAP,  Effort  on  Test,  and 
Product 

3 

.1730** 

.0040 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Tiredness  During 

Test 

2 

. 1727** 

.0074 

Regular  ASAP,  Tiredness  During 

Test,  and  Product 

3 

. 1773** 

.0046 

Regular  ASAP 

1 

. 1653** 

Regular  ASAP  and  Computer  Use 

2 

. 1654** 

.0001 

Regular  ASAP,  Computer  Use,  and 
Product 

3 

. 1674** 

.0020 
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Table  5  (Continued) 


Measures 

n 

R 

R  Increase 

Regular  ASAP 

i 

. 1653** 

Regular  ASAP  and  Paragraph 

Comprehens ion 

2 

. 1688** 

.0035 

Regular  ASAP,  Paragraph 

Comprehension  and  Product 

3 

.1707** 

.0019 

Directionally  Weighted  Item 

Score — Double  Standardized 

1 

. 1581** 

Directionally  Weighted  Item 

Score  and  Correlation  Between 
Subject’s  and  Sample’s 

Latencies 

2 

.1641** 

.0060 

Directionally  Weighted  Item 

Score,  Correlation  Between 

Subject’s  and  Sample’s  Latencies, 
and  Product 

3 

. 1642** 

i 

.0001 

Directionally  Weighted  Item 
Score--Double  Standardized 

1 

. 1581** 

Directionally  Weighted  Item 

Score  and  Interquartile  Range 

2 

. 1662** 

.0081 

Directionally  Weighted  Item 

Score,  Interquartile  Range,  and 
Product 

3 

.1662** 

.0000 

Directionally  Weighted  Item 
Score--Double  Standardized 

1 

. 1581** 

Directionally  Weighted  Item 

Score  and  Interquartile 

Deviations  of  3.5  or  More 

2 

. 1595** 

.0014 

Directionally  Weighted  Item 

Score,  Interquartile  Deviations 
of  3.5  or  More,  and  Product 

3 

. 1619** 

.0024 

Table  5  (Continued) 
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Measures 

n 

R 

R  Increase 

Directionally  Weighted  Item 

Score — Double  Standardized 

1 

. 1581** 

Directionally  Weighted  Item 

Score  and  Difference  Between 

Median  Interquartile  Deviation 
for  Subject  and  Sample 

2 

. 1583** 

.0002 

Directionally  Weighted  Item 

Score,  Difference  Between 

Median  Interquartile  Deviation 
for  Subject  and  Sample,  and 

Product 

3 

. 1602** 

.0019 

Directionally  Weighted  Item 

Score — Double  Standardized 

1 

. 1581** 

Directionally  Weighted  Item 

Score  and  Items  Changed 

2 

.1596** 

.0015 

Directionally  Weighted  Item 

Score,  Items  Changed,  and 

Product 

3 

. 1647** 

.0051 

Directionally  Weighted  Item 
Score--Double  Standardized 

1 

.1581** 

Directionally  Weighted  Item 

Score  and  Maximum  Changes 

Per  Item 

2 

. 1658** 

.0077 

Directionally  Weighted  Item 

Score,  Maximum  Changes  Per 

Item,  and  Product 

3 

. 1659** 

.0001 

Directionally  Weighted  Item 
Score — Double  Standardized 

Directionally  Weighted  Item 
Score  and  Effort  on  Test 

Directionally  Weighted  Item 
Score,  Effort  on  Test,  and 
Product 


1  .1581-** 

2  . 1623**  .0042 


. 1659** 


3 


.0036 
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Table  5  (Continued) 


Measures 

n 

R 

R  Increase 

Directionally  Weighted  Item 

Score — Double  Standardized 

i 

. 1581** 

Directionally  Weighted  Item 

Score  and  Tiredness  During 

Test 

2 

. 1660** 

.0079 

Directionally  Weighted  Item 

Score  and  Tiredness  During 

Test,  and  Product 

3 

.1700** 

.0040 

Directionally  Weighted  Item 
Score--Double  Standardized 

1 

.1581** 

Directionally  Weighted  Item 

Score  and  Computer  Use 

2 

.1581** 

.0000 

Directionally  Weighted  Item 

Score,  Computer  Use,  and 

Product 

3 

.1621** 

.0040 

Nondirectionally  Weighted  Item 

Score — Double  Standardized 

1 

. 1547** 

Nondirectionally  Weighted  Item 

Score  and  Correlation  Between 
Subject’s  and  Sample’s  Latencies 

2 

. 1619** 

.0072 

Nondirectionally  Weighted  Item 

Score,  Correlation  Between 

Subject’s  and  Sample’s  Latencies, 
and  Product 

3 

. 1620** 

.0001 

Nondirectionally  Weighted  Item 
Score--Double  Standardized 

1 

. 1547** 

Nondirectionally  Weighted  Item 

Score  and  Interquartile  Range 

2 

. 1638** 

.0091 

Nondirectionally  Weighted  Item 

Score,  Interquartile  Range,  and 
Product 

3 

. 1640** 

.0002 

Table  5  (Continued) 
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Measures 

n 

R 

R  Increase 

Nondirectionally  Weighted  Item 

Score — Double  Standardized 

i 

.1547** 

Nondirectionally  Weighted  Item 

Score  and  Interquartile 

Deviations  of  3.5  or  More 

2 

. 1564** 

.0017 

Nondirectionally  Weighted  Item 

Score,  Interquartile 

Deviations  of  3.5  or  More, 
and  Product 

3 

.1636** 

.0072 

Nondirectionally  Weighted  Item 

Score — Double  Standardized 

1 

.1547** 

Nondirectionally  Weighted  Item 

Score  and  Difference  Between 

Median  Interquartile  Deviation 
for  Subject  and  Sample 

2 

.1550** 

.0003 

Nondirectionally  Weighted  Item 

Score,  Difference  Between 

Median  Interquartile  Deviation 
for  Subject  and  Sample,  and  Product 

3 

.1551** 

.0001 

Nondirectionally  Weighted  Item 
Score--Double  Standardized 

1 

. 1547** 

Nondirectionally  Weighted  Item 

Score  and  Items  Changed 

2 

. 1564** 

.0017 

Nondirectionally  Weighted  Item 

Score,  Items  Changed,  and  Product 

3 

. 1607** 

.0043 

Nondirectionally  Weighted  Item 
Score--Double  Standardized 

1 

. 1547** 

Nondirectionally  Weighted  Item 

Score  and  Maximum  Changes  Per 

I  tern 

2 

. 1627** 

.0080 

Nondirectionally  Weighted  Item 

Score,  Maximum  Changes  Per 

Item,  and  Product 

3 

. 1629** 

.0002 
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Table  5  (Continued) 

Measures 

n 

R 

R  Increase 

Nondirectionally  Weighted  Item 
Score-~Double  Standardized 

i 

.  1547** 

Nondirectionally  Weighted  Item 

Score  and  Effort  on  Test 

2 

.1585** 

.0038 

Nondirectionally  Weighted  Item 

Score,  Effort  on  Test,  and 

Product 

3 

. 1603** 

.0018 

Nondirectionally  Weighted  Item 
Score--Double  Standardized 

1 

.1547** 

Nondirectionally  Weighted  Item 

Score  and  Tiredness  During  Test 

2 

.1629** 

.0082 

Nondirectionally  Weighted  Item 

Score,  Tiredness  During  Test, 
and  Product 

3 

.1698** 

.0069 

Nondirectionally  Weighted  Item 

Score — Double  Standardized 

1 

. 1547** 

Nondirectionally  Weighted  Item 

Score  and  Computer  Use 

2 

. 1547** 

.0000 

Nondirectionally  Weighted  Item 

Score,  Computer  Use,  and 

Product 

3 

. 1572** 

.0025 

Nondirectionally  Weighted  Item 
Score--Double  Standardized 

1 

. 1547** 

Nondirectonally  Weighted  Item 

Score  and  Paragraph  Comprehension 

2 

. 1582** 

.0035 

Nnndirectionally  Weighted  Item 

Score,  Paragraph  Comprehension, 
and  Product 

3 

1 596** 

.0014 

Table  5  (Continued) 
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Measures 

n 

R 

R  Increase 

Item  Subgroup  Scores — Double 
Standardized 

10 

.  1810** 

Item  Subgroup  Scores  and 

Correlation  Between  Subject’s 
and  Sample’s  Latencies 

11 

. 1870** 

.0060 

Item  Subgroup  Scores,  Correlation 
Between  Subject’s  and  Sample’s 
Latencies,  and  Product 

21 

.2037** 

.0167 

Item  Subgroup  Scores — Double 
Standardized 

10 

. 1810** 

Item  Subgroup  Scores  and 

Interquartile  Range 

11 

. 1887** 

.0077 

Item  Subgroup  Scores, 

Interquartile  Range, 
and  Product 

21 

.2079** 

.0192 

Item  Subgroup  Scores--Double 
Standardized 

10 

. 1810** 

Item  Subgroup  Scores  and 

Interquartile  Deviations  of 

3.5  or  More 

11 

. 1820** 

.0010 

Item  Subgroup  Scores,  Interquartile 
Deviations  of  3.5  or  More,  and 
Product 

21 

. 2079** 

.0259 

Item  Subgroup  Scores--Double 
Standardized 

10 

. 1810** 

Item  Subgroup  Scores  and 

Difference  Between  Median 
Interquartile  Deviation  for 

Subject  and  Sample 

11 

. 1812** 

.0002 

Item  Subgroup  Scores,  Difference 
Between  Median  Interquartile 
Deviation  for  Subject  and 

Sample,  and  Product 

21 

. 1992** 

.0180 

Table  5  (Continued) 
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Measures 

n 

R 

R  Increase 

Item  Subgroup  Scores--Double 
Standardized 

10 

.1810** 

Item  Subgroup  Scores  and  Items 
Changed 

11 

. 1826** 

.0016 

Item  Subgroup  Scores,  Items 

Changed,  and  Product 

21 

.2087** 

.0261 

Item  Subgroup  Scores — Double 
Standardized 

U) 

. 1810** 

Item  Subgroup  Scores  and 

Maximum  Changes  Per  Item 

11 

.1877** 

.0067 

Item  Subgroup  Scores,  Maximum 

Changes  Per  Item,  and  Product 

21 

.  1971** 

.0094 

Item  Subgroup  Scores — Double 
Standardized 

10 

.  1810** 

Item  Subgroup  Scores  and  Effort 
on  Test 

11 

.  1850** 

.0040 

Item  Subgroup  Scores,  Effort  on 

Test,  and  Product 

21 

.2051** 

.0201 

Item  Subgroup  Scores--Double 
Standardized 

10 

. 1810** 

Item  Subgroup  Scores  and 

Tiredness  During  Test 

11 

. 1870** 

.0060 

Item  Subgroup  Scores,  Tiredness 
During  Test,  and  Product 

21 

.2111** 

.0241 

Item  Subgroup  Scores--Double 
Standardized 

10 

. 1810** 

Item  Subgroup  Scores  and 

Computer  Use 

11 

.1810** 

.0000 

Item  Subgroup  Scores,  Computer 

Use,  and  Product 

21 

.  1937** 

.0127 
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Table  5  (Continued) 


Measures 

n 

R 

R  Increase 

Item  Subgroup  Scores--Double 
Standardized 

10 

.  1810** 

Item  Subgroup  Scores  and 

Paragraph  Comprehension 

11 

.  1842** 

.0032 

Item  Subgroup  Scores,  Paragraph 
Comprehension,  and  Product 

21 

.1892** 

.0050 

Directionally  Weighted  Item 

Score--S ingle  Standardized 

1 

. 1609** 

Directionally  Weighted  Item 

Score  and  Correlation  Becwcen 
Subject’s  and  Sample’s  Latencies 

2 

.1670** 

.0061 

Directionally  Weighted  Item 

Score,  Correlation  Between 

Subject’s  and  Sample’s 

Latencies,  and  Product 

3 

. 1671** 

.0001 

Directionally  Weighted  Item 
Score--Single  Standardized 

1 

. 1609** 

Directionally  Weighted  Item 

Score  and  Interquartile  Range 

2 

. 1691** 

.0082 

Directionally  Weighted  Item 

Score,  Interquartile  Range, 
and  Product 

3 

. 1691** 

.0000 

Directionally  Weighted  Item 
Score--Single  Standardized 

1 

. 1609** 

Directionally  Weighted  Item 

Score  and  Interquartile 

Deviations  of  3.5  or  More 

2 

. 1624** 

.0015 

Directionally  Weighted  Item 

Score,  Interquartile 

Deviations  of  3.5  or  More, 
and  Product 

3 

. 1651** 

.0027 
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Table  5  (Continued) 


Measures 


n 


R  R  Increase 


Directionally  Weighted  Item 

Score — Single  Standaradized  1 

Directionally  Weighted  Item 
Score  and  Difference  Between 
Median  Interquartile  Deviation 
for  Subject  and  Sample  2 

Directionally  Weighted  Item 

Score,  Difference  Between  Median 

Interquartile  Deviation  for 

Subject  and  Sample,  and  Product  3 


. 1609** 


.1612**  .0003 


.1629**  .0017 


Directionally  Weighted  Item 

Score — Single  Standardized 

1 

. 1609** 

Directionally  Weighted  Item 

Score  and  Items  Changed 

2 

. 1624** 

.0015 

Directionally  Weighted  Item 

Score,  Items  Changed,  and 

Product 

3 

. 1697** 

.0073 

Directionally  Weighted  Item 
Scores--Single  Standardized 

1 

. 1609** 

Directionally  Weighted  Item 

Score  and  Maximum  Changes 

Per  Item 

2 

. 1685** 

.0076 

Directionally  Weighted  Item 

Score,  Maximum  Changes  Per 

Item,  and  Product 

3 

. 1685** 

.0000 

Directionally  Weighted  Item 
Score--Single  Standardized 

1 

. 1609** 

Directionally  Weighted  Item 

Score  and  Effort  on  Test 

Directionally  Weighted  Item 

Score,  Effort  on  Test,  and 

2 

. 1650** 

.  004 1 

Product 

3 

. 1682** 

.0032 
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Table  5  (Continued) 


Measures 

n 

R 

R  Increase 

Directionally  Weighted  Item 

Score — Single  Standardized 

1 

.  1609** 

Directionally  Weighted  Item 

Score  and  Tiredness  During  Test 

2 

. 1686** 

.0077 

Directionally  Weighted  Item 

Score,  Tiredness  During  Test, 
and  Product 

3 

. 1732** 

.0046 

Directionally  Weighted  Item 

Score — Single  Standardized 

1 

.  1609** 

Directionally  Weighted  Item 

Score  and  Computer  Use 

2 

. 1610** 

.0001 

Directionally  Weighted  Item 

Score,  Computer  Use,  and 

Product 

3 

.1659** 

.0049 

Nondirectionally  Weighted  Item 
Score--Single  Standardized 

1 

. 1630** 

Nondirectionally  Weighted  Item 

Score  and  Correlation  Between 
Subject’s  and  Sample’s  Latencies 

2 

.  1699** 

.0069 

Nondirectionally  Weighted  Item 

Score,  Correlation  Between 

Subject’s  and  Sample’s 

Latencies,  and  Product 

3 

.  1707** 

.0008 

Nondirectionally  Weighted  Item 
Score--Single  Standardized 

1 

. 1630** 

Nondirectionally  Weighted  Item 

Score  and  Interquartile  Range 

2 

. 1711** 

.0081 

Nondirectionally  Weighted  Item 

Score,  Interquartile  Range, 
and  Product 

3 

. 1711** 

.  0000 
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Table  5  (Continued) 


Measures 


ti 


R  R  Increase 


Nondirectionally  Weighted  Item 
Score--S ingle  Standardized 

1 

. 1630** 

N’ondirectionally  Weighted  Item 

Score  and  Interquartile 

Deviations  of  3.5  or  More 

2 

. 1646** 

.0016 

N'ondirectionally  Weighted  Item 

Score,  Interquartile  Deviations 
of  3.5  or  More,  and  Product 

3 

. 1708** 

.0062 

N'ondirectionally  Weighted  Item 

Score — Single  Standardized  1 

N’ondirectionally  Weighted  Item 
Score  and  Difference  Between 
Median  Interquartile  Deviation 
for  Subject  and  Sample  2 

N'ondirectionally  Weighted  Item 
Score,  Difference  Between 
Median  Interquartile  Deviation 
for  Subject  and  Sample,  and 
Product  3 


.  1630** 


.1633**  .0003 


.1643**  .0010 


N'ondirectionally  Weighted  Item 
Score--S ingle  Standardized 

1 

. 1630** 

N'ondirectionally  Weighted  Item 

Score  and  Items  Changed 

2 

. 1647** 

.0017 

N'ondirectionally  Weighted  Item 

Score,  Items  Changed,  and 

Produc t 

3 

. 1717** 

.0070 

o n d i root i o n a 1 1 v  Weighted  Item 

So  o  re  — S  ingle  Standardized 

1 

. 1630** 

ond i rec t iona 1 1 y  Weighted  Item 

Score  and  Maximum  Changes 

Per  Item 

2 

. 1704** 

o 

o 

'--J 

ond i rec t iona 1 ly  Weighted  Item 

Score,  Maximum  Changes  Per 

Item,  and  Product 

3 

. 1706** 

.0002 
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Table  5  (Continued) 


Measures 

n 

R 

R  Increase 

Nondirectionally  Weighted  Item 

Score — Single  Standardized 

i 

. 1630** 

Nondirectionally  Weighted  Item 

Score  and  Effort  on  Test 

2 

. 1667** 

.0037 

Nondirectionally  Weighted  Item 

Score,  Effort  on  Test,  and 

Product 

3 

. 1704** 

.0037 

Nondirectionally  Weighted  Item 
Score--Single  Standardized 

1 

. 1630** 

Nondirectionally  Weighted  Item 

Score  and  Tiredness  During 

Test 

2 

. 1703** 

.0073 

Nondirectionally  Weighted  Item 

Score,  Tiredness  During  Test, 
and  Product 

3 

. 1747** 

.0044 

Nondirectionally  Weighted  Item 

Score — Single  Standardized 

1 

. 1630** 

Nondirectionally  Weighted  Item 

Score  and  Computer  Use 

2 

. 1630** 

.  0000 

Nondirectionally  Weighted  Item 

Score,  Computer  Use,  and 

Product 

3 

. 1635** 

.  0003 

Nondirectionally  Weighted  Item 

Score--S ingle  Standardized 

1 

. 1630** 

Nondirectionally  Weighted  Item 

Score  and  Paragraph  Comprehension 

2 

. 1669** 

.0039 

Nondirectionally  Weighted  Item 

Score,  Paragraph  Comprehension, 
and  Product 

3 

o 

CO 

vO 

.  00  1  1 

Table  5  (Continued) 
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Measures 

n 

R 

R  Increase 

Item  Subgroup  Scores--Single 
Standardized 

10 

. 1813** 

Item  Subgroup  Scores  and 

Correlation  Between  Subject’s 
and  Sample’s  Latencies 

11 

. 1867** 

.0054 

Item  Subgroup  Scores,  Correlation 
Between  Subject’s  and  Sample's 
Latencies,  and  Product 

21 

. 2094** 

.0227 

Item  Subgroup  Scores--Single 
Standardization 

10 

.1813** 

Item  Subgroup  Scores  and 

Interquartile  Range 

11 

. 1891** 

.0078 

Item  Subgroup  Scores, 

Interquartile  Range,  and  Product 

21 

.2357** 

.0466* 

Item  Subgroup  Scores--S ingle 
Standardization 

10 

.1813** 

Item  Subgroup  Scores  and 

Interquartile  Deviations  of 

3.5  or  More 

11 

. 1824** 

.0011 

Item  Subgorup  Scores, 

Interquartile  Deviations  of 

3.5  or  More,  and  Product 

21 

. 2063** 

.0239 

Item  Subgroup  Scores--Single 
Standardized 

Item  Subgroup  and  Difference 
Between  Median  Interquartile 
Deviation  for  Subject  and 
Sample 

Item  Subgroup,  Difference  Between 
Median  Interquartile  Deviation 
for  Subject  and  Sample,  and 
Produc  t. 


10  . 1813** 


11  . 1815**  .0002 


.  2096** 


21 


.0281 
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Table  5  (Continued) 


Measures 

n 

R 

R  Increase 

Item  Subgroup  Scores — Single 
Standardized 

10 

. 1813** 

Item  Subgroup  Scores  and 

Items  Changed 

11 

. 1826** 

.0013 

Item  Subgroup  Scores,  Items 

Changed,  and  Product 

21 

. 1937** 

.0111 

Item  Subgroup  Scores — Single 
Standardized 

10 

. 1813** 

Item  Subgroup  Scores  and 

Maximum  Changes  Per  item 

11 

.1883** 

.0070 

Item  Subgroup  Scores,  Maximum 

Changes  Per  Item,  and  Product 

21 

.2012 

.0129 

Item  Subgroup  Scores — Single 
Standardized 

10 

.1813** 

Item  Subgroup  Scores  and 

Effort  on  Test 

11 

. 1843** 

.0030 

Item  Subgroup  Scores,  Effort  on 

Test,  and  Product 

21 

. 2078** 

.0235 

Item  Subgroup  Scores — Single 
Standardized 

10 

. 1813** 

Item  Subgroup  Scores  and 

Tiredness  During  Test 

11 

. 1877** 

.0064 

Item  Subgroup  Scores,  Tiredness 
During  Test,  and  Product 

21 

.2128** 

.0251 

Item  Subgroup  Scores--S ingle 
Standardized 

10 

. 1813** 

Item  Subgroup  Scores  and 

Computer  Use 

11 

.  1813** 

.0000 

Item  Subgroup  Scores,  Computer 

Use,  and  Product 

21 

.2218** 

.0405 

Table  5  (Continued) 
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Measures 

n 

R 

R  Increase 

Item  Subgroup  Scores--Single 
Standardized 

10 

. 1813** 

Item  Subgroup  Scores  and 

Paragraph  Comprehension 

11 

.  1844** 

.0031 

Item  Subgroup  Scores, 

Paragraph  Comprehension, 
and  Product 

21 

. 1981** 

.0137 

*p  <.05;  **p  <.01 
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Figure  Captions 

Figure  1.  Internal-consistency  reliability  of  double-standardized  item 
subgroup  scores. 

Figure  2.  Internal-consistency  reliability  of  single-standardized  item 
subgroup  scores. 

Figure  3.  Predictive  validity  of  double-standardized  item  subgroup 
scores . 

Figure  4.  Predictive  validity  of  single-standardized  item  subgroup 


scores . 
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Washington,  DC  20370-2000 

Assistant  for  Training  Technology 
and  Human  Factors 
Office  of  the  DCNO(MPT)  (Op-llBl) 
Department  of  the  Navy 
Washington,  DC  203502000 

Head,  Military  Compensation 
Policy  Branch 

Office  of  the  DCNO(MPT)  (OP- 134) 
Department  of  the  Navy 
Washington,  DC  20370  2000 

Deputy  Director  Military 
Personnel  Policy  Division 
Office  of  the  DCNOfMPT)  (OP-13B) 
Department  of  the  Navy 
Washington,  DC  203702000 

Head, 

Family  Support  Program  Branch, 

OP  156 

1300  Wilson  BtaL.  Room  828 
Arlington,  VA  22209 

Head,  Manpower,  Personnel, 
and  Training  Branch 
Office  of  the  CNO  (OP-813) 

4A47H  The  Pentagon 
Washington.  DC  203501000 

Assistant  for  Manpower 
and  Training 

Office  of  the  CNO  (OP-9R7H) 

5B6R3.  The  Pentagon 
Washington.  DC  20350 

Dr.  Judith  Orasanu 
Basic  Research  Office 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Jesse  Oria nsky 
Institute  for  Defense  Analyses 
1801  N.  Beauregard  St 
Alexandria.  VA  22311 


Assistant  for  Training  and 
Per-onnel  Technolo© 

Office  of  the  Under  Secretary  of 
Defense  for  Research  &  Engrg. 
3D  129,  The  Pentagon 
Washington,  DC  20301-3080 

Dr.  Peter  J.  Pashley 
Educational  Testing  Service 
Rosedale  Road 
Princeton,  NJ  08541 

Wayne  M.  Patience 
American  Council  on  Education 
GED  Testing  Service,  Suite  20 
One  Dupont  Circle,  NW 
Washington,  DC  20036 

Dr.  James  Paulson 
Department  of  Psychology 
Portland  State  University 
P.O.  Box  751 
Portland,  OR  97207 

Director,  Center  for  Personnel 
Security  Research 
Suite  E,  Building  455 
99  Pacific  Street 
Monterey,  CA  93640-2481 

Dept,  of  Administrative  Sciences 
Code  54 

Naval  Postgraduate  School 
Monterey.  CA  939-13-5026 

Department  of  Operations  Research. 

Naval  Postgraduate  School 
Monterey,  CA  93940 

Dr.  Mart  D.  Reckase 
ACT 

P.  O.  Box  168 
Iowa  City,  l  A  52243 

Dr.  Malcolm  Ree 
AFHRIVMOA 
Brooks  AFB.  TX  78235 

Mr.  Steve  Reiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  Rrver  Road 
Minneapolis,  MN  55455-0344 

Dr.  Carl  Ross 
CNETPDCD 
Building  90 

Great  I  .akes  NTC,  1L  60088 
Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

Dr.  Eduardo  Salas 
Human  Factors  Division  (Code  712) 
Naval  Training  Systems  Center 
Orlando,  FL  32813-7100 

Dr.  Fumiko  Samejima 
Department  of  Psychology 
Universiry  of  Tennessee 
310B  Austin  Peay  Bldg 
Knoxville,  TN  379!<yoonn 

Mr.  Drew  Sands 

NPRDC  Okie  62 

San  Diego.  CA  92152  6800 

Dr.  Walter  Schneider 
learning  RAD  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 
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Lowell  Schoer 

Psychological  A  Quantitative 
Foundations 
College  of  Education 
University  of  Iowa 
Iowa  City,  !A  522-12 

l>r,  Mary  Schratz 
‘1100  Parkside 
Carlsbad,  CA  Q2008 

Dr.  Dan  Segal! 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

Mr.  Robert  Semmea 
N218  Elliott  Hall 
Department  of  Psychology 
University  of  Minnesota 
Minneapolis,  MN  55455 

Dr.  Robin  Sbeaty 
Illinois  Stale  Wtuer  Survr*' 

Room  149 
2201  GrifTith  Dr. 

Champaign,  IL  61820 

Dr.  Kazan  Shigemnsu 
7-9-24  Kugenuma-Kaigan 
Fujisawa  251 
JAPAN 

Dr.  Randall  Shumaker 
Naval  Research  Laboratory 
Code  5510 

4555  Overlook  Avenue,  S.W. 
Washington,  DC  20575-5000 

Dr.  1 1.  Wallace  Sinaiko 
Manpower  Research 
and  Advisory  Services 
Smithsonian  Institution 
801  North  Pitt  Street,  Suite  120 
Alexandria,  VA  22314-1713 

Dr.  Richard  E.  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  91305 

Dr.  Richard  C  Sorensen 
Navy  Personnel  RAD  Center 
San  Diego,  CA  92152-6800 

Dr.  Judy  Spray 
ACr 

P.O.  Box  168 
Iowa  City,  IA  52243 

Dr.  Martha  Stocking 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  Peter  StolofT 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.O.  Box  16268 
Alexandria,  VA  22302-0268 

Dr.  William  Stout 
University  of  Illinois 
Department  of  Statistics 
101  lllini  liall 

South  Wright  Sl 
Champaign,  IL  61820 

Dr.  I.awrence  J,  Strieker 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  Hariharan  Swamifialhan 
laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  f duration 
University  of  Massachusetts 
Amherst,  MA  01003 


Mr.  Brad  Sympson 

Navy  Person nd  RAD  Center 

Code-62 

San  Diego,  CA  92152-6800 

Dr.  John  Tangney 
AFOSR/NU  BUg,  410 
Bolling  APB,  DC  Vmi-tM 

Dr.  Kikumi  Tatsuoka 
Educational  Testing  Service 
Mail  Stop  03-T 
Princeton,  NJ  0654 1 

Dr.  Maurice  Tauuoka 
Educational  Testing  Service 
Mall  Stop  03-T 
Princeton.  NJ  «541 

Dr.  David  Thissen 
Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

Mr.  Thomas  J.  Thomas 
Johns  Hopkins  University 
Department  of  Psychology 
Charles  A  34tb  Street 
Baltimore,  MD  21218 

Mr.  Gary  Thomasson 
University  of  Illinois 
Educational  Psychology 
Champaign,  IL  61820 

Dr.  Perry  W.  Thorn  dyke 
FMC  Corporation 
Central  Engineering  Labs 
1205  Coleman  Avenue,  Box  580 
Santa  Clara,  CA  95052 

Dr,  Robert  Tsutakawa 
University  of  Missouri 
Department  of  Statistics 
222  Math.  Sciences  Bldg 
Columbia,  MO  65211 

Dr,  l>edyard  Tucker 
University  of  Illinois 
Department  of  Psychology 
603  E.  Daniel  Street 
Champaign,  IL  61820 

Dr.  David  Vale 
Assessment  Systems  Corp. 

2233  University  Avenue 
Suite  440 

Sl  Paul,  MN  55114 

Dr.  Prank  L  Vidno 
Navy  Personnel  RAD  Center 
San  Diego,  CA  92152-6800 

Dr.  Howard  Wainer 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr’  Michael  T.  Waller 
University  of  Wacom  In- Milwaukee 
Educational  Piwbology  Department 
Box  413 

Milwaukee,  W1  53201 

Dr.  Mlng-Mei  Wing 
Educational  Testing  Service 
Mail  Stop  03-T 
Princeton,  NJ  08541 

Dr,  Thomas  A  Warm 
PAA  Academy  AAC934D 
P.O.  Box  25082 
Oklahoma  City,  OK  73125 

Dr.  Brian  Waien 
HumRRO 
1100  S.  Washlnpon 
Alexandria,  VA  22314 


Dr.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455-0344 

Dr.  Ronald  A  Weinman 
Box  146 

Carmel,  CA  93921 

Major  John  Welsh 
AFHRL/MOAN 
Brooks  AFB,  TX  78223 

Dr,  Douglas  Wetzel 
Code  51 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152-6800 

Dr.  Rand  R.  Wilcox 
University  of  Southern 
California 

'.v.nartment  of  Psychology 
Lw  Angeles,  CA  90069-1061 

German  Military  Representative 
ATTN:  Wolfgang  Wildgrube 
Strcitkraefteamt 
D-5300  Bonn  2 
4000  Brandywine  Street,  NW 
Washington,  DC  20016 

Dr.  David  Wiley 
School  of  Education 
Northwestern  University 
Evanston,  IL  60201 

Dr.  Charles  Wilkins 
Navy  Personnel  RAD  Center 
Code  13 

San  Diego,  CA  92152 

Dr.  Bruce  Williams 
Department  of  Educational 
Psychology 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Kent  E.  Wiliams 

InsL  for  Simulation  and  Training 

University  of  Central  Florida 

P.O.  Box  25000 

Orlando,  FL  32816-0544 

Dr.  Mark  Wilson 
School  of  Education 
University  of  California 
Berkeley,  CA  94720 

Dr.  Hilda  Wing 

Federal  Aviation  Administration 
800  Independence  Ave,  $W 
Washington,  DC  20591 

‘Mr.  John  H.  Wolfe 
Navy  Personnel  RAD  Center 
San  Diego,  CA  92152-6800 

Dr.  Oeorge  Wong 
Bloalatlitlc*  Laboratory 
Memorial  Sloan-Kettering 
Cancer  Center 
1275  York  Avenue 
New  York,  NY  10021 

Dr.  Wallace  Wulfeck,  III 
Navy  Personnel  RAD  Center 
Code  51 

San  Diego,  CA  92152-6800 

Dr.  Reniaro  Yamamoto 
02-T 

Educational  Testing  Service 
Roiedale  Road 
Princeton,  NJ  06541 
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Dr.  Wendy  Yen 
CTB/McGraw  Hill 
Del  Monte  Research  Part 
Monterey,  CA  93940 

Dr.  Joseph  L.  Young 
National  Science  Foundation 
Room  320 
1800  G  Street,  N.W. 
Washington,  DC  20550 

Mr.  Anthony  R.  Zara 
National  Council  of  State 
Boards  of  Nursing,  Inc 
625  North  Michigan  Avenue 
Suite  1544 
Chicago,  IL  60611 

Dr.  Steven  Zomctzer 
Office  of  Naval  Research 
Code  114 
800  N.  Quincy  Sl 
Arlington.  VA  22217-5000 


